Allegedly, the term "smoke testing" comes from
the plumbing
industry. When talking about electronics, it means "turn it on and see if it catches fire".
See Wikipedia for more.
Which plugins are tested?
The goal is to test every plugin in the WordPress.org plugin directory. In practice, we've tested about
98% of those plugins at least once. Some plugins cannot be tested due to technical constraints or because
they're missing important details like "Version" headers.
Does an "ok" result mean that the plugin is guaranteed to work?
Not quite. This is just a very basic automated test. There are many types of bugs that it can't catch.
Also, we only test plugins in one particular environment (WordPress version + PHP version + server
settings). If your server is very different, you might still run into compatibility issues.
Treat the test result as a starting point, not a final judgement.
Does a "failure" mean that the plugin is broken and unusable?
It suggests that there's something wrong, but it doesn't always mean that the plugin is broken.
Here's why:
The testing tool isn't smart enough to reliably distinguish between critical bugs and
scary-but-harmless errors. We err on the side of caution. For example, we treat every database error
as a serious problem. In practice, some of them are minor issues that may have no visible
effect on your site.
The testing tool doesn't read the documentation. The plugin description might clearly
say that you must install another plugin first or that you need to set up your site in a particular
way, but the test ignores all of that. The test installs each plugin on a fresh WordPress
site, using the default settings. If the plugin crashes, we treat that as a test failure.
These tests use a headless browser. There are a few plugins that trigger errors in this browser
but that work perfectly fine in a "real" browser. These false positives are rare.