Supported `transformers` versions are not clear

[abdulfatir]

There are no clear version ranges for transformers specified in the setup files. This leads to people running into issues like pinokiocomputer/pinokio#1006. Unclear version ranges makes it very hard for other packages to depend on peft. Could you clarify which versions of transformers are supported by different peft versions? In general, it would great if the version ranges are specified in setup.py or pyproject.toml so that dependents can safely select supported versions based on their requirements.

[BenjaminBossan]

Thanks for bringing this. We understand the concern and agree that having this knowledge would be helpful. However, getting this information is an almost impossible ask. The combinations of PEFT methods, transformers models, and use cases is already gigantic. If we now multiply this with the number of transformers releases, we end up with an impossibly big number to test. Right now, we test the latest transformers release as well as the main branch and our CI is already stretching the limits.

There is in fact no combination of transformers and PEFT versions where everything works, always something new comes up that may break and sometimes there are also regressions. It can easily happen that PEFT method X works with model Y in transformers version v1.2.3 and v1.2.5 but not v1.2.4. So what can a PEFT user generally do in this situation?

1. Our recommendation is to always use the latest releases of PEFT and transformers, unless something else prevents it. This has the highest chance of working.
2. If, for some reason, you have to use an older transformers version, I'd still try the latest PEFT version first, as we generally are backwards compatible (there is PEFT code to maintain BC with transformers < v4.33 -- going much further back than that is not a good idea).
3. If that doesn't work, try a PEFT version from a similar time as the transformers release.

Note that with the upcoming transformers v5 release, the minimum PEFT version will in fact be 0.18.0. On the PEFT side, we have no plans of dropping support for older transformers versions.

[abdulfatir]

Hello @BenjaminBossan, Thanks a lot for your reply!

I definitely understand the problem for existing versions. However, moving forward I think it would be great to invest some time into this to explicitly specify version ranges. This is because your recommendations unfortunately only work for direct users of peft but not for packages depending on it. For stable packages like https://github.com/autogluon/autogluon and https://github.com/amazon-science/chronos-forecasting (where we plan to integrate peft as a dependency), we cannot really force users to use a specific or the latest version of different packages and have to allow a reasonable range of supported package versions.

[BenjaminBossan]

Thanks for understanding. We're in a very similar boat when it comes to this, as we also want to support a broad range of versions in PEFT but dependencies can run into the same issue of incompatible versions. But I still don't see a viable way to declare "working" version ranges.

To give you a concrete example, we know that the prefix tuning PEFT method had issues with Gemma models. Say, it was version 4.52 to 4.56. What would be the correct thing to do in PEFT? Exclude these versions from being supported? But then, PEFT would work absolutely fine if the user trains a Llama model or uses LoRA. If we only declare support for versions where "everything" works, that would be none. If we turn a blind eye to this specific incompatibility, where do we stop?

If you know that for your project, you absolutely need prefix tuning to work with Gemma, you can still set the version ranges accordingly. But we cannot do that in PEFT, as it inconvenience all those users who don't need this specific combination. I know it's annoying to have to figure that out in the first place, but as mentioned, it's impossible for use to list all working and non-working combinations.

I think the main thing we can do is to provide a minimum transformers version that we don't support. For the sake of the argument, let's say it's 4.32. Do you think that would help in your project? My guess is barely, but if you disagree, we can discuss this idea.