Removing duplicate words from a text file?

[Random Man]

I have a 2.8 gb text file, and I need all the duplicate words removed...

 

Big text files: http://download.cnet.com/Large-Text-File-Viewer/3000-2379_4-90541.html

As for removing duplicates, this code would do it

Dim arrLines() As String = Split(Textfile, vbCrLf)
Dim lstLines As List(Of String) = arrLines.ToList()

For i As Integer = 0 To Lines.Length - 1

    For j As Integer = 0 To Lines.Length - 1

        If lstLines(i) = lstLines(j) Then
            
            lstLines.RemoveAt(j)

        End If        

    Next

Next

You then just save the content of lstLines into a new textfile.

 

[Random Man]

Big text files: http://download.cnet.com/Large-Text-File-Viewer/3000-2379_4-90541.html

As for removing duplicates, this code would do it

Dim arrLines() As String = Split(Textfile, vbCrLf)
Dim lstLines As List(Of String) = arrLines.ToList()

For i As Integer = 0 To Lines.Length - 1

    For j As Integer = 0 To Lines.Length - 1

        If lstLines(i) = lstLines(j) Then
            
            lstLines.RemoveAt(j)

        End If        

    Next

Next

You then just save the content of lstLines into a new textfile.

Alright. Thanks.