txt file duplicate line remover python

Annoying Aardvark answered on September 19, 2020 Popularity 9/10 Helpfulness 9/10

answer txt file duplicate line remover python

related python delete duplicate lines in file

related Python remove duplicate lines from a text file

txt file duplicate line remover python

Comment

lines_seen = set() # holds lines already seen

with open("file.txt", "r+") as f:
    d = f.readlines()
    f.seek(0)
    for i in d:
        if i not in lines_seen:
            f.write(i)
            lines_seen.add(i)
    f.truncate()

Popularity 9/10 Helpfulness 9/10 Language python

Source: stackoverflow.com

Tags: file line python txt

Link to this answer
Share Copy Link

Contributed on Sep 19 2020

Annoying Aardvark

0 Answers Avg Quality 2/10

Closely Related Answers

python delete duplicate lines in file

Comment

with open("file.txt", "r") as txt_file:
  new_data = list(set(txt_file))
  return new_data

Popularity 9/10 Helpfulness 6/10 Language python

Source: Grepper

Tags: file

Link to this answer
Share Copy Link

Contributed on Nov 30 2020

Supermavster

0 Answers Avg Quality 2/10

Python remove duplicate lines from a text file

Comment

import hashlib
def main():
    input_file = "in.txt"
    output_file = "out.txt"
    
    completed_hash = set()
    output_file = open(output_file, "w")
    
    for line in open(input_file,"r"):
        hashValue = hashlib.md5(line.strip().encode('utf-8')).hexdigest()
        
        if hashValue not in completed_hash:
            output_file.write(line)
            completed_hash.add(hashValue)
            
    output_file.close()


if __name__ == "__main__":
    main()

Popularity 8/10 Helpfulness 3/10 Language python

Source: www.codevscolor.com

Tags: file lines

Link to this answer
Share Copy Link

Contributed on Mar 16 2022

Arif Muxtarov

0 Answers Avg Quality 2/10

txt file duplicate line remover python

Contents

More Related Answers

txt file duplicate line remover python

Closely Related Answers

python delete duplicate lines in file

Python remove duplicate lines from a text file

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.