Fixit 2: Meta’s next-generation auto-fixing linter

• Fixit is useless! Lengthy dwell Fixit 2 – the newest model of our open-source auto-fixing linter.
• Fixit 2 permits builders to effectively construct customized lint guidelines and carry out auto-fixes for his or her codebases.
• Fixit 2 is accessible immediately on PyPI.

Python is likely one of the hottest languages in use at Meta. Meta’s manufacturing engineers (PEs) are specialised software program engineers (SWEs) who give attention to reliability, effectivity, and scalability. They work on numerous initiatives, together with debugging manufacturing providers, rewriting inefficient libraries, orchestrating venture deployments at scale, or capability planning and scheduling. And Python is usually one of many first instruments that PEs attain for, because it gives fast improvement, straightforward to learn syntax, and a large array of open supply libraries.

Meta’s Python Language Basis group — a hybrid group of each PEs and conventional SWEs — helps personal and preserve the infrastructure and tooling for Python at Meta. The group helps engineers, information scientists, researchers, and anybody else at Meta utilizing Python to get their work completed.

One of many methods we accomplish that is constructing instruments that allow Python builders to put in writing higher, and extra dependable code extra effectively. This contains instruments like automatic formatting and import sorting that eradicate tedium, or linters that information engineers towards maintainable code with fewer bugs.

This 12 months, we’ve been constructing a brand new linter, Fixit 2, designed from the bottom as much as make builders extra environment friendly and succesful, each in open supply initiatives and the various panorama of our inner monorepo. At Meta, we’re utilizing Fixit 2 with a couple of early adopters, and plan to roll it out to the remainder of our monorepo quickly. However any developer can use it to carry out auto-fixing extra effectively and make sooner enhancements to their very own codebases.

Why a brand new linter? (why not X?)

There are a selection of fantastic linters within the Python ecosystem, a lot of which have a big neighborhood of third-party plugins offering a various array of lint guidelines. We’ve used Flake8 internally at Meta since 2016, and it has been very profitable in serving to builders cut back bugs and maintain a clear codebase. The favored flake8-bugbear plugin was even created by Łukasz Langa (creator of Black, PSF developer-in-residence, and launch supervisor for Python 3.8 and three.9) whereas working at Meta (then Fb), as a house for extra opinionated lint guidelines that we might each use internally and share with the remainder of the Python developer neighborhood. 

We even have a lot of inner plugins constructed by numerous groups, and Flake8 permits them to put in writing and allow customized lint guidelines immediately within the codebase with out getting sign-off from a central gatekeeper, and with out ready for a brand new deployment of Flake8 to roll out.

However whereas Flake8 has lengthy been a cornerstone of our linting answer, it additionally has some tough edges. Writing new lint guidelines requires constructing total plugins (every claiming a portion of the “namespace” for error codes) and encourages builders to construct sophisticated plugins masking a number of lessons of errors. When these lint errors are discovered, Flake8 can solely level to a line and column quantity the place it occurred, however has no method of suggesting adjustments to the developer a listing of lint outcomes, leaving them in a state of trial and error to search out adjustments that make the linter comfortable. Additionally, Flake8 makes use of the stdlib ast module, making it unable to parse future syntax options and forcing builders to attend for instruments to improve earlier than they will use the shiny new hotness.

There are options to Flake8 after all, however a lot of them endure from a number of drawbacks: 

• A scarcity of assist for “native” in-repo plugins or customized lint guidelines. 
• Restricted or no assist for hierarchical configuration for various initiatives inside a monorepo.
• No possibility for auto-fixes when errors are discovered.
• Sluggish efficiency on giant codebases.

Whereas a few of these options aren’t essential, a very powerful for developer effectivity is providing auto-fixes – automated urged adjustments that may fulfill the lint rule. This takes the guesswork out of utilizing a linter, and permits customers to rapidly assessment and settle for these adjustments when doable, eliminating the necessity to re-run the linter till the code is lastly clear. Combining these auto-fixes with in-repo, customized lint guidelines gives a degree of tailor-made code high quality enhancements that’s onerous to beat.

Sadly, even Fixit, the auto-fixing linter that we constructed for Instagram and open sourced, didn’t assist native lint guidelines or hierarchical configuration – core necessities for our monorepo that’s dwelling to 1000’s of initiatives, a lot of that are themselves open supply initiatives with their very own distinct wants for linting and CI. We obtained many requests from builders to assist Fixit in our monorepo, however there have been sufficient hurdles that we had been solely in a position to assist a small set of safety lint guidelines, decreasing the direct advantages to our Python codebase. 

Meet Fixit 2

After discussions with different groups, particularly within the quickly rising AI/ML area, we thought of our choices and determined upon a partial rewrite of Fixit. We deliberately designed the brand new model with an open source-first mindset, whereas incorporating the wants and necessities of our personal monorepos and open supply initiatives from day one.

The framework and linting engine could be rebuilt from the bottom up whereas leaving the core design of lint guidelines largely untouched. The brand new system gives a hierarchical configuration primarily based on the TOML format; assist for native, in-repo lint guidelines much like Flake8; and a a lot improved CLI and API for integration with different instruments and automation. 

Fixit itself builds on prime of one other Instagram open supply venture, LibCST, a concrete syntax tree for Python with a tree and node API following the patterns of the ast module in the usual library. The “concrete” a part of CST implies that LibCST contains each a part of the supply file within the ensuing tree after parsing, together with whitespace, feedback, and formatting components which are ignored by the ast module. That is what permits Fixit (and different instruments we constructed, like µsort) to securely modify supply information, with out utilizing common expressions or the danger of manufacturing damaged syntax, and gives the muse for Fixit to supply auto-fixes urged by the lint guidelines themselves.

Writing a brand new lint rule may be completed with lower than a dozen strains of code, and check instances are outlined inline. You possibly can even place it proper subsequent to the code that it will likely be linting:

# teambread/guidelines/hollywood.py
import fixit
import libcst
class HollywoodName(fixit.LintRule):
    VALID = [...] # no lint errors right here
    INVALID = [...] # dangerous code samples right here
    def visit_SimpleString(self, node: libcst.SimpleString):
        if node.worth in ('"Paul"', "'Paul'"):
            self.report(node, "It is underbaked!")

Suggesting auto-fixes for the consumer is as straightforward as together with a brand new CST node when reporting an error:

def visit_SimpleString(self, node: libcst.SimpleString):
    if node.worth in ('"Paul"', "'Paul'"):
        new_node = libcst.SimpleString('"Mary"')
        self.report(node, new_node)

Enabling this new rule inside the venture’s codebase may be completed with a easy config change:

# teambread/sourdough/fixit.toml
[tool.fixit]
allow = [".rules.hollywood"]

Now we are able to run our linter in opposition to our venture:

# teambread/sourdough/baker.py
identify = "Paul"
print(f"hiya identify!")

$ fixit lint --diff sourdough/baker.py
sourdough/baker.py@7:11 HollywoodName: It is underbaked! (has autofix)
--- a/baker.py
+++ b/baker.py
@@ -6,3 +6,3 @@
def principal():
-    identify = "Paul"
+    identify = "Mary"
    print(f"hiya identify")
🛠️  1 file checked, 1 file with errors, 1 auto-fix accessible 🛠️
[1]

The `lint` command solely exhibits errors and urged adjustments. If we use the `repair` command, we are able to apply these urged adjustments again to the codebase:

$ fixit repair --automatic sourdough/baker.py
sourdough/baker.py@7:11 HollywoodName: It is underbaked! (has autofix)
🛠️  1 file checked, 1 file with errors, 1 auto-fix accessible, 1 repair utilized 🛠️

Now that our auto-fixes have been utilized, we are able to affirm that the venture is now clear and lint-free:

$ fixit lint sourdough/baker.py
🧼 1 file clear 🧼

When working Fixit 2 with auto-fixing lint guidelines, any code that triggers the lint rule is a chance to get an automated alternative, bettering the codebase with much less effort from the developer. Utilized extra broadly, Fixit 2 may even be used as a instrument to enact sweeping codemods in opposition to a big codebase, whereas leaving a lint rule in place to deal with any matching code sooner or later.

Attempt Fixit 2 

Fixit 2 is offered immediately on PyPI. You possibly can set up and check Fixit 2 with pip set up fixit. 

We’ve a roadmap with plans for future enhancements and options, and a wealthy set of documentation and user guides that can assist you get began with Fixit 2 in your personal initiatives or repositories. We hope it proves helpful in your initiatives, and we look ahead to hearing your feedback!