• P03 Locke@lemmy.dbzer0.comEnglish
    57·
    27 days ago

    Bad research is still bad. The main paper is 9 pages with wide margins, outside of references. This was a few college students publishing their CS research assignment. The “Limitations” section is rather telling:

    Another interesting avenue opened by this work is how to improve the automatic generation of useful context files.

    Okay, so they already admitted that they started with useless context files, which sabotages their own test.

    You know how you fix useless context files? You iterate until they are useful context files!

    This whole research paper is XKCD 385, except with AGENTS.md files.

    • fruitcantfly@programming.dev
      21·
      26 days ago

      I haven’t read the paper in dept, but these are deeply unserious reasons for disparaging it.

      Bad research is still bad. The main paper is 9 pages with wide margins, outside of references. This was a few college students publishing their CS research assignment

      The length of this paper entirely within the norm for scientific publications. In fact, depending on the type of paper, they can be much shorter. For example, Nature expects submissions at around 2500 words or around 4300 words, depending on the format, while this paper is around 4800 words long (excluding references and SI).

      The authors also obviously not “a few college students publishing their CS research assignment”: Gloaguen, Mündler, Raychev, Müller, and Vechev.

      The “Limitations” section is rather telling:

      Another interesting avenue opened by this work is how to improve the automatic generation of useful context files.

      Okay, so they already admitted that they started with useless context files, which sabotages their own test.

      You know how you fix useless context files? You iterate until they are useful context files!

      The entire point of the study was to see what kind of context files were useful and which were useless, which according to the authors was not something that had been done before:

      A widespread practice in software development is to tailor coding agents to repositories using context files, such as AGENTS.md. Although this practice is strongly encouraged by agent developers, there is currently no rigorous investigation into whether such context files are actually effective for real-world tasks. In this work, we study this question and evaluate coding agents’ task completion performance in two complementary settings

      They found that,

      while instructions in the context files are well followed by coding agents, repository overviews, although popular and recommended by model providers, are not helpful.

      So when you say that they “admitted that they started with useless context files”, you are referring to the results of the study. Complaining that they didn’t base their study on the conclusions of that very study is just absurd

      • P03 Locke@lemmy.dbzer0.comEnglish
        1·
        22 days ago

        The entire point of the study was to see what kind of context files were useful and which were useless,

        The title of the Pivot-to-AI article is: “Your AGENTS.md file doesn’t actually do anything”

        These two statements don’t jive.

        So when you say that they “admitted that they started with useless context files”, you are referring to the results of the study. Complaining that they didn’t base their study on the conclusions of that very study is just absurd

        This is the difference between “we found a lot of useless AGENTS.md files” and “AGENTS.md files cannot possibly be useful at all”. The second is the argument the article is making. The first is the argument the research paper is making, which like most things on the Internet, is effectively a tautology. Most of the internet is useless, and most people are too criminally stupid to even operate a computer.