Text file word list

 sed -e 's/[^[:alpha:]]/ /g' text_to_analize.txt | tr 'n' " " |  tr -s " " | tr " " 'n'| tr 'A-Z' 'a-z' | sort | uniq -c | sort -nr | nl 

This command makes the following:

1. Substitute all non alphanumeric characters with a blank space.
2. All line breaks are converted to spaces also.
3. Reduces all multiple blank spaces to one blank space
4. All spaces are now converted to line breaks. Each word in a line.
5. Translates all words to lower case to avoid ‘Hello’ and ‘hello’ to be different words
6. Sorts de text
7. Counts and remove the equal lines
8. Sorts reverse in order to count the most frequent words
9. Add a line number to each word in order to know the word posotion in the whole

For example if I want to analize the first Linus Torvald message:

From: torvalds@klaava.Helsinki.FI (Linus Benedict Torvalds)
Newsgroups: comp.os.minix Subject: What would you like to see most in
minix? Summary: small poll for my new operating system Message-ID:
<1991Aug25.205708.9541@klaava.Helsinki.FI> Date: 25 Aug 91 20:57:08
GMT Organization: University of Helsinki

Hello everybody out there using minix –

I’m doing a (free) operating system (just a hobby, won’t be big and
professional like gnu) for 386(486) AT clones. This has been brewing
since april, and is starting to get ready. I’d like any feedback on
things people like/dislike in minix, as my OS resembles it somewhat
(same physical layout of the file-system (due to practical reasons)
among other things).

I’ve currently ported bash(1.08) and gcc(1.40), and things seem to
work. This implies that I’ll get something practical within a few
months, and I’d like to know what features most people would want. Any
suggestions are welcome, but I won’t promise I’ll implement them 🙂

Linus (torvalds@kruuna.helsinki.fi)

PS. Yes – it’s free of any minix code, and it has a multi-threaded fs.
It is NOT protable (uses 386 task switching etc), and it probably
never will support anything other than AT-harddisks, as that’s all I
have :-(.

I create a file named linus.txt, I paste the content and then I write in the console:

sed -e 's/[^[:alpha:]]/ /g' linus.txt | tr 'n' " " |  tr -s " " | tr " " 'n'| tr 'A-Z' 'a-z' | sort | uniq -c | sort -nr | nl 

The out put would be:

 1        7 i
 2        5 to
 3        5 like
 4        5 it
 5        5 and
 6        4 minix
 7        4 a
 8        3 torvalds
 9        3 of
10        3 helsinki
11        3 fi
12        3 any
13        2 would
14        2 won
15        2 what
16        ...

If you want to visualize only the first 20 words:

sed -e 's/[^[:alpha:]]/ /g' text_to_analize.txt | tr 'n' " " |  tr -s " " | tr " " 'n'| tr 'A-Z' 'a-z' | sort | uniq -c | sort -nr | nl | head -n 20

It’s important to note that the command tr ‘A-Z’ ‘a-z’ doesn’t suport UTF-8 yet, so that in foreign languages the word APRÈS would be translated as aprÈs.

def MakeWordList():
    infile=open(('possible.rtf'),'r')
    whole = infile.readlines()
    infile.close()

    L=[]
    for line in whole:
        word= line.split(' ')
        if word not in L:
            L.append(word)
            if len(word) in range(5,100):
                L.append(word)
                if not word.endswith('xx'):
                    L.append(word)
                    if word == word.lower():
                        L.append(word)
    print L

MakeWordList()

3 Answers

if word not in L and len(word) >= 5 and not 'xx' in word and word.islower():
    L.append(word)

    if word not in L and len(word) >= 5:
        if not 'xx' in word and word.islower():
            L.append(word)

def MakeWordList():
    with open('possible.rtf','r') as f:
        data = f.read()
    return set([word for word in data if len(word) >= 5 and word.islower() and not 'xx' in word])

Главная : Расширения файлов : .wordlist File

The wordlist program that reads a text file and makes an alphabetical list of all the words in that file. The list of words is output to another file. Improve the program so that it also keeps track of the number of times that each word occurs in the file. Write two lists to the output file. The first list contains the words in alphabetical order.The number of times that the word occurred in the file should be listed along with the word. Then write a second list to the output file in which the words are sorted according to the number of times that they occurredin the files. The word that occurred most often should be listed first.

Code for Word list program that reads a text file and makes an alphabetical list of all the words in that file. The list of words is output to another file in Java

import java.io.*;

class cntwords
{
          
      publicstaticvoid main(String args[]) throws IOException
      {
    
    try
    {
      FileReader fr = new FileReader("data.txt");
      BufferedReader br = new BufferedReader(fr);
      String str;

      while((str = br.readLine()) != null)
            countwords(str);
      fr.close();
    }
    catch(FileNotFoundException e)
     {
      System.out.println("File Not Found");
    }
    catch(IOException e)
        {
      System.out.println("Exception : " + e);
    } 
      }
    
      staticvoid countwords(String st)
      {
    int len,i,totwords=1,j=-1,k=0,freqarr[],temp;
    int cnt=0,flag=0;
    len = st.length();
    char ch[];
    String starr[],st1[];
        
    for(i=0;i<len-1;i++)
    {
       if(st.charAt(i) != ' ' && st.charAt(i+1) == ' ')
          totwords++;
    }
     
        starr = new String[totwords+1];
        freqarr = newint[totwords];
    st1 = new String[totwords];

    for(i=0;i<len-1;i++)
    {
        if(st.charAt(i) != ' ' && st.charAt(i+1) == ' ')
        {
           ch = newchar[(i+1) - j];
           st.getChars(j+1,i+1,ch,0);
           starr[k++] = new String(ch);
           j = i+1;
        }    
    }

    ch = newchar[(i+1) - j];    
        st.getChars(j+1,i+1,ch,0);
        starr[k] = new String(ch);
    k=0;

    for(i=0;i<totwords;i++)
        {
        temp = 1;
       flag = 0;
           for(j=i+1;j<=totwords;j++)
       {
          if(starr[i].equals(starr[j]))
            temp++;
        }
       freqarr[k] = temp;

           for(cnt=0;cnt<k;cnt++)
           {
          if(starr[i].equals(st1[cnt]))
                 flag = 1;
           }
       if(flag == 0)
             st1[k++] = starr[i];    
    }   
                
    for(i=0;i<k;i++)
           System.out.println(st1[i] + "has count : " + freqarr[i]);
    writedata(st1,freqarr,k);
      }
    
      staticvoid writedata(String arr[],int intarr[],int len)
      {
          try
          {

       FileOutputStream fos = new FileOutputStream("res.txt");
       FileOutputStream fos1 = new FileOutputStream("res1.txt");            
       DataOutputStream dos = new DataOutputStream(fos);
       int i,j,val;
           String temp;
           for(i=0;i<len;i++)
       {
              dos.writeChars(arr[i]); 
          dos.writeChars(Integer.toString(intarr[i]));
          dos.writeChar(' ');
       }
           fos.close();
           dos = new DataOutputStream(fos1);
    
       for(i=0;i<len;i++)
           {
              for(j=i+1;j<len;j++)
              {
              if(intarr[i] < intarr[j])
                  {
                     val = intarr[j];
                     intarr[j] = intarr[i];
                     intarr[i] = val;
                     temp = arr[j];
                     arr[j] = arr[i];
                     arr[i] = temp;
                  }
              }
           }

       for(i=0;i<len;i++)
       {
              dos.writeChars(arr[i]); 
          dos.writeChars(Integer.toString(intarr[i]));
          dos.writeChar(' ');
       }    
       fos1.close();
      }
      catch(IOException e)
          {
           System.out.println("Exception : " + e);
    
      }
      
      }   
}