The same holds true in Python: Read the files with TextIO and follow with a Map operation that splits the lines into records.
This, of course, only works if you don't have newlines within your records. In that case, you may need to use a DoFn that takes as input a each filename and reads the entire file (e.g. using the standard library csv parsers), emitting the records (possibly followed by a Reshuffle), e.g.
| beam.Create([list of filenames])
| beam.FlatMap(lambda path: csv.reader(open(path)))