Abstract page for arXiv paper 2604.03136: StoryScope: Investigating idiosyncrasies in AI fiction

  • Wataba@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    3
    ·
    4 hours ago

    Find me an option that works like GDocs keeping a running history of writing and editing, and I’ll consider it.

    I have receipts for the integrity of my writing. I dont need some shitty tool to validate it.

    • zqps@sh.itjust.works
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 hour ago

      Some moron is gonna vibe code a script that spoofs this on the surface. Yea it’ll be detectable, but not detected in most cases.

    • commonmarmoset@reddthat.com
      link
      fedilink
      English
      arrow-up
      1
      ·
      4 hours ago

      I love that docs receipt feature and agree with you. Great call.

      Forgive me being well-meaningly obnoxious, but make sure to get receipts outside of the Google ecosystem too. As a professional writer/editor myself, I’ve heard horror stories about folks losing access to an account, and therefore the receipts.

      I am also skeptical of these kinds of tools.

      • Holytimes@sh.itjust.works
        link
        fedilink
        English
        arrow-up
        1
        ·
        3 hours ago

        Never trust any cloud storage that you do not personally have access to cold back ups of.

        This is data storage 101!

  • Zen_Shinobi@lemmy.world
    link
    fedilink
    English
    arrow-up
    3
    arrow-down
    3
    ·
    3 hours ago

    Just look for the em dash. Anytime someone says they have been using it before AI is a liar.

  • AnAmericanPotato@programming.dev
    link
    fedilink
    English
    arrow-up
    54
    ·
    19 hours ago

    93.2% macro-F1 for human vs. AI detection and 68.4% macro-F1 for six-way authorship attribution

    This isn’t what I’d call “reliable”.

    I’m also not impressed with their methodology, which is heavily based on Gemini.

    To generate mirrored AI stories, we reverse-engineer writing prompts from each human story by prompting Gemini 2.5 Flash (Gemini Team, 2025a) to infer the underlying premise

    That’s too many steps removed from anything you’d encounter in the wild. They’re not even testing against human-prompted output.

    And then they use Gemini again to analyze all the stories. Relying on proprietary cloud models for the core of your analysis is like building on sand.

    • MangoCats@feddit.it
      link
      fedilink
      English
      arrow-up
      6
      arrow-down
      1
      ·
      12 hours ago

      Next up: AI adapts to evade detection by last week’s researchers’ detectors.

      • Kevin@programming.dev
        link
        fedilink
        English
        arrow-up
        34
        ·
        18 hours ago

        Percentages are somewhat misleading here. That’s around 2 in 25, enough to falsely accuse a student in a fairly standard classroom for example.

        I remember somebody mentioning a quote along the lines of “A 95% success rate sounds good, until you become the 1 in 20 that it kills”

        • MangoCats@feddit.it
          link
          fedilink
          English
          arrow-up
          1
          arrow-down
          1
          ·
          12 hours ago

          In matters of life and death, the alternative choices are always important to consider…

          5% chance of dying vs a 99% chance of living a horrible life of immobility and pain?

          We all die eventually.

          • Holytimes@sh.itjust.works
            link
            fedilink
            English
            arrow-up
            1
            ·
            3 hours ago

            Getting your degree or being kicked out and banned from your college and still requires to pay back the loans. IS life or death.

            It literally can determine the quality of life of a person for the next 30-40 years. If not forever. This is NOT something to take lightly.

      • Damarus@feddit.org
        link
        fedilink
        English
        arrow-up
        12
        arrow-down
        1
        ·
        18 hours ago

        That’s not good enough. Just a thought experiment: Only 7% of humans died because their work got flagged incorrectly

        • Allah@piefed.worldOP
          link
          fedilink
          English
          arrow-up
          3
          arrow-down
          29
          ·
          18 hours ago

          this is just spherical cow fallacy combined with purity culture slop

          no methodology is perfect, it just needs to be practically valid, using ai != killing people for using ai

          unless you are for killing people who use AI, which wouldn’t surprise me one bit about Anti-AI horde

          • darkmarx@lemmy.world
            link
            fedilink
            English
            arrow-up
            22
            ·
            17 hours ago

            Tell that to the student who gets expelled because they’re writing was falsely identified as AI and then takes their own life.

            Tell that to the person falsely identified with facial recognition and is arrested and loses their job.

            Using AI to detect AI is as bad as using AI for any important decision.

            This isn’t hypothetical. It happens. People will lose their livelihood or die because of it. 7 in 100 is not accurate enough when any penalty is applied to detection.

            • MangoCats@feddit.it
              link
              fedilink
              English
              arrow-up
              1
              arrow-down
              1
              ·
              12 hours ago

              The key is for the detector to be the screener, innocent until proven guilty, and a screener tool with 7% failure rate is not proof.

          • MangoCats@feddit.it
            link
            fedilink
            English
            arrow-up
            2
            ·
            12 hours ago

            Back when I was in Uni poor grades were considered a lifelong sentence to poverty…

  • palordrolap@fedia.io
    link
    fedilink
    arrow-up
    17
    ·
    19 hours ago

    … which will be fed into the next generation of LLMs in order to stop sounding quite so much like LLMs.

    • MangoCats@feddit.it
      link
      fedilink
      English
      arrow-up
      1
      ·
      12 hours ago

      Already, you can tell 'em “stop using em-dashes and other AI tells” — and they will.

  • Exec@pawb.social
    link
    fedilink
    English
    arrow-up
    21
    ·
    20 hours ago

    I wonder how does this stand to writers who have learnt English as a second language

    • Waterpumpee@lemmus.org
      link
      fedilink
      English
      arrow-up
      2
      arrow-down
      4
      ·
      16 hours ago

      At this point, unless you write prose, i’d certainly use AI to improve formulations. See it as proof readers.

    • REDACTED@infosec.pub
      link
      fedilink
      English
      arrow-up
      4
      ·
      5 hours ago

      I can’t tell if you posted this unironically, but in any case this is just bad AI slop

    • sbbq@lemmy.zip
      link
      fedilink
      English
      arrow-up
      36
      arrow-down
      2
      ·
      21 hours ago

      I can’t even imagine what dickhead is saying there. Is he implying that the mean real people are bullying the sensitive AI babies?

      • AbouBenAdhem@lemmy.world
        link
        fedilink
        English
        arrow-up
        25
        arrow-down
        2
        ·
        21 hours ago

        Since Ben Garrison is the one human whose output most closely resembles AI, I imagine this is him having an existential crisis.

        • davidgro@lemmy.world
          link
          fedilink
          English
          arrow-up
          19
          arrow-down
          1
          ·
          edit-2
          19 hours ago

          You’re saying an actual human called that thing both “pen & ink” and a “typewrite”?

          … Also labeled the clearly working lamp as “power outage”

          … And doesn’t understand how USB works

          I don’t buy it. Even if Ben did real drawing in the past, he clearly isn’t now. Edit: Or as a reply pointed out, the signature could be fake too with Ben not involved.

          • Rose@slrpnk.net
            link
            fedilink
            English
            arrow-up
            2
            arrow-down
            1
            ·
            4 hours ago

            You’re saying an actual human called that thing both “pen & ink” and a “typewrite”?

            It’s Ben Garrison! Dude can’t get an erection unless he labels every single goddamn thing. While being barely coherent. He’s been doing this for ages!

          • AnAmericanPotato@programming.dev
            link
            fedilink
            English
            arrow-up
            10
            arrow-down
            1
            ·
            20 hours ago

            Is this real? Or is it a fake comic making fun of Ben Garrison? It’s hard to tell. I’d expect better even from him, which is saying a lot.

          • frongt@lemmy.zip
            link
            fedilink
            English
            arrow-up
            7
            arrow-down
            2
            ·
            19 hours ago

            I dont think this is real. Definitely AI generated, very unlikely to be his actual work.

            • edgemaster72@lemmy.world
              link
              fedilink
              English
              arrow-up
              1
              ·
              17 hours ago

              I agree, reverse image search isn’t coming up with anything for this one, but it works on other comics from his site. Seems to be an imitation.

      • ThePowerOfGeek@lemmy.world
        link
        fedilink
        English
        arrow-up
        10
        arrow-down
        1
        ·
        21 hours ago

        Why did I think he had died? Probably getting him mixed up with Scott Adams.

        I agree with you, it’s a typical example of a Garrison cartoon. It’s so boringly obvious on a surface level that you can’t help but assume there’s some idiotic alt-right subtext.

      • Allah@piefed.worldOP
        link
        fedilink
        English
        arrow-up
        3
        arrow-down
        24
        ·
        21 hours ago

        what? bro the joke is that ai writers are not real writers, it’s not that deep.