Change the logic to use the tokenizer directly. This allows
adding more unit tests to check the validty of the tokenizer
itself.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Message-ID: <2672257233ff73a9464c09b50924be51e25d4f59.1773074166.git.mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <66e6320a4d5ad9730c1c0ceea79b5021e90c66c6.1773770483.git.mchehab+huawei@kernel.org>