Powershell to clean up a text file

arti · 2021-12-01T19:36:48.000Z

Hi there,

I’m requesting help on powershell script to clean up a input file and write to an output file.

Contents of the input.txt file are…

abcd
cdef
[“abcd”]
["ab
cd"]

[“efgh”]

xyz

Expected OUTPUT.txt file

abcd
cdef
[“abcd”]
[“abcd”]

[“efgh”]

xyz

Wanted to suggest the criteria to clean up the file is
If a line contains start square bracket but does not end with square bracket, then concatenate the next line to current line.

Looking forward and appreciate your help towards script.

Thanks.

The below is missing few lines and im not sure where im missing.

$file = "c:\junk\test.txt"
 $BufferedLine = $null
    
 Get-Content $file |
     ForEach-Object {
         if ($_ -match "^[^[].*[^]]$") {
             # line doesn't begin with "[" or end with "]"
             $_                                  # -- line is okay, just return it
         }
         elseif ($_ -match "^\[.*\]$") {
             # line begins with "[" and ends with "]"
             $_                                  # -- line is okay, just return it
         }
         elseif ($_ -match "^\[.*[^]]$") {
             # line begins with "[" but doesn't end with "]"
             $BufferedLine = $_                  # remember line as beginning
         }
         elseif ($_ -match "^[^[].*\]$") {
             # line doesn't begin with "[" but ends with "]"
             if ($BufferedLine) {                # AND there's a preceeding line awaiting closure
                 $BufferedLine += $_             # concatenate with contents of previous line
                 $BufferedLine                   # return completed line
                 $BufferedLine = $null           # and forget the value
             }
    
         }
     }
 if ($BufferedLine) {
     $BufferedLine                           # return the last line of necessary
 }

fddde38a-f6ac-453e-8097-e6b11e74a434-input.txt (50 Bytes)

brianswales · 2021-12-01T19:39:43.000Z

To quote a previous comment to this type of post:

To quote everyone in the PowerShell forums here, we are not a script writing service, but we’re happy to help figure out why something you’ve tried isn’t working. What have you tried so far, and what parts are you having trouble with? Don’t forget to use the “insert code” button (looks like a </> in the WYSIWYG editor) to paste any code you have.

Neally · 2021-12-01T19:40:29.000Z

Welcome

Arti:

I’m requesting help

We request you to show what you have tried, or where you’re stuck?

We’re happy to help, but not a scriptwriting service.

If you post code, please use the ‘Insert Code’ button. Please and thank you!

PLEASE READ BEFORE POSTING! Read if you're new to the PowerShell forum! Programming & Development

Hi, and welcome to the PowerShell forum! Don’t apologize for being a “noob” or “newbie” or “n00b.” There’s just no need – nobody will think you’re stupid, and the forums are all about asking questions. Just ask! Use a descriptive subject. Don't say "Need help" or "PowerShell Help", actually summarize what the problem is. It helps the rest of us keep track of which problem is which. Don’t post massive scripts. We’re all volunteers and we don’t have time to read all that, nor will we copy, past…

192033ab-bb8f-4032-88a5-8e2313af0344-codebutton_small.png

Neally Bot

DoctorDNS · 2021-12-01T19:46:34.000Z

You can do this. First use Get-content to read the file in to an array. Then for each line in the array, check the line using a regular expression switch to pick up normal lines vs lines starting with ‘[’ etc. Then at you iterate, write the corrected line to a new array. Might be 15 lines of code.

arti · 2021-12-01T21:48:18.000Z

I have posted the input file and what i tried. The code is missing some lines in output. not sure where im missing. Thanks.

Evan7191 · 2021-12-02T02:41:25.000Z

Arti:

Hi there,

I’m requesting help on powershell script to clean up a input file and write to an output file.

Contents of the input.txt file are…

abcd
cdef
[“abcd”]
["ab
cd"]

[“efgh”]

xyz

Expected OUTPUT.txt file

abcd
cdef
[“abcd”]
[“abcd”]

[“efgh”]

xyz

Wanted to suggest the criteria to clean up the file is
If a line contains start square bracket but does not end with square bracket, then concatenate the next line to current line.

Looking forward and appreciate your help towards script.

Thanks.

The below is missing few lines and im not sure where im missing.
$file = "c:\junk\test.txt"
 $BufferedLine = $null
    
 Get-Content $file |
     ForEach-Object {
         if ($_ -match "^[^[].*[^]]$") {
             # line doesn't begin with "[" or end with "]"
             $_                                  # -- line is okay, just return it
         }
         elseif ($_ -match "^\[.*\]$") {
             # line begins with "[" and ends with "]"
             $_                                  # -- line is okay, just return it
         }
         elseif ($_ -match "^\[.*[^]]$") {
             # line begins with "[" but doesn't end with "]"
             $BufferedLine = $_                  # remember line as beginning
         }
         elseif ($_ -match "^[^[].*\]$") {
             # line doesn't begin with "[" but ends with "]"
             if ($BufferedLine) {                # AND there's a preceeding line awaiting closure
                 $BufferedLine += $_             # concatenate with contents of previous line
                 $BufferedLine                   # return completed line
                 $BufferedLine = $null           # and forget the value
             }
    
         }
     }
 if ($BufferedLine) {
     $BufferedLine                           # return the last line of necessary
 }

I ran this:

$file = Get-Content -Path "c:\junk\test.txt" -Raw
$file = $file -ireplace '(?<match1>\[[^\]]*)\r\n(?<match2>[^\]]*\])','${match1}${match2}'
$file | Out-File -FilePath "C:\Junk\Results.txt"

I got this output:

abcd
cdef
["abcd"]
["abcd"]

["efgh"]
xyz

arti · 2021-12-02T14:10:32.000Z

Thank you. Super cool Evan… Cant thank you enough you are like a ninja to get this in almost 1 line.

My bad. I’m attaching a revised file to make it easier. The intent is to search for [" (start square bracket with quotes) anywhere in a line which does not contain close square bracket with quotes "]

Rest of the lines should not be impacted.

For some reason i was able to attach input file but browser does not allow to attach expectedoutput file.

Expected o/p

"
[
[abcd,
cdef
[“abcd”],
,[“spdf”]
[“abcd”]

[“efgh”],
xyz
],
]

5f824ad0-f975-4019-9b4c-348e719fed4a-input.txt (77 Bytes)

Evan7191 · 2021-12-02T14:45:16.000Z

I ran my code against the text file that you uploaded and got this output:

"
[
[abcd,
cdef["abcd"],
,["spdf"]
["abcd"]

["efgh"],
xyz
],
]

That looks like your expected output to me, unless I am missing something.

Arti:

The intent is to search for [" (start square bracket with quotes) anywhere in a line which does not contain close square bracket with quotes "]

That’s basically what my RegEx does, except that it does not specify quotation marks. Quotation marks could be added to the RegEx pretty easily. For example:

$file = $file -ireplace '(?<match1>\[\"[^\]]*)\r\n(?<match2>[^\]]*\"\])','${match1}${match2}'

This yielded the same output that I listed above.

arti · 2021-12-02T20:45:28.000Z

Hi Evan,

I tested and came across a scenario where I see some special characters in line 9,10,11,12 in the attached file. Not sure if you are able to see this msg since i marked as answered. When you get a chance you think you could help me in handling this. I couldnt find what these special characters are.

I ran the below command.

$file = Get-Content -Path "c:\junk\pinput.txt" -Raw
$file = $file -ireplace '(?<match1>\[\"[^\]]*)\r\n(?<match2>[^\]]*\"\])','${match1}${match2}'
$file 

#| Out-File -FilePath "C:\Junk\Results.txt"

2274cf5e-e093-4ef5-8c5e-f1ed27130c9c-pinput.txt (245 Bytes)

Evan7191 · 2021-12-02T21:23:13.000Z

Arti:

Hi Evan,

I tested and came across a scenario where I see some special characters in line 9,10,11,12 in the attached file. Not sure if you are able to see this msg since i marked as answered. When you get a chance you think you could help me in handling this. I couldnt find what these special characters are.

I ran the below command.
$file = Get-Content -Path "c:\junk\pinput.txt" -Raw
$file = $file -ireplace '(?<match1>\[\"[^\]]*)\r\n(?<match2>[^\]]*\"\])','${match1}${match2}'
$file 

#| Out-File -FilePath "C:\Junk\Results.txt"

Usually special characters like that are caused by the encoding. Out-File uses unicode by default. You can use the -Encoding parameter to specify UTF8 or another encoding.

learn.microsoft.com

Out-File (Microsoft.PowerShell.Utility) - PowerShell

The Out-File cmdlet sends output to a file. It implicitly uses PowerShell's formatting system to write to the file. The file receives the same display representation as the terminal. This means that the output may not be ideal for programmatic...

arti · 2021-12-07T15:27:34.000Z

Hi Evan,

$file = $file -ireplace ‘(?["[^]])\r\n(?[^]]"])’,‘${match1}${match2}’ (existing logic looks for line containing with [" and not ending with ]"

I would like to modify the above logic a little since i came across a scenario Image attached.

This time i would like to look for line containing [" and does not end with ], then concatenate line 1 and 2.

I tried the following but i think im missing something.

$file = $file -ireplace ‘(?["[^]])\r\n(?[^]]],)’,‘${match1}${match2}’

f1bf5a49-7add-4ca9-ad77-a4803d5e49e3-cleanup.png800×451 37 KB

I tried postprocessing the above output for lines with encoding issue. For those lines with single quotes

get-content C:\temp\pinput.txt -Encoding UTF8 | Set-Content c:\temp\poutput.txt - works as expected.

i tried get-content $file -Encoding UTF8 | Set-Content c:\temp\poutput.txt - it failed. im sure im doing it wrong.

arti · 2021-12-07T15:39:49.000Z

Forgot to attach input file. One is text file and other is js file but limitation to attach js file.

19c7d86d-380b-467b-951c-5e51468ac5dc-pinput.txt (322 Bytes)

arti · 2021-12-07T15:41:13.000Z

File ext needs to be renamed as .js as another example input file.

47d1e1f3-b110-444d-815e-f2dec9555f5b-jsfile.txt (443 Bytes)