array in hash table having duplicates

spiceuser-evezb · 2021-04-23T20:07:32.000Z

I am trying to avoid duplicate going in array ( $clients[$client].pool , $clients[$client].db and $clients[$client].dd )in hash table. but still i am getting duplicate in below piece of code , any idea ??

Thank you

if($client -notin $clients.Keys){
$clients[$client] = @{
‘pool’ = @()}
$clients[$client][‘db’] = @{
‘db’ = @()}
$clients[$client][‘dd’] = @{
‘dd’ = @()}
$clients[$client].pool = "$pool "
$clients[$client].db = "$db "
$clients[$client].dd = "$dd "
}else{
if ("$pool " -notin $clients[$client].pool) {$clients[$client].pool += "$pool "}
if ("$db " -notin $clients[$client].db) {$clients[$client].db += "$db "}
if ("$dd " -notin $clients[$client].dd ) {$clients[$client].dd += "$dd "}
}

Neally · 2021-04-23T20:13:52.000Z

Welcome.

If you post code, please use the ‘Insert Code’ button. Please and thank you!

PLEASE READ BEFORE POSTING! Read if you're new to the PowerShell forum! Programming & Development

Hi, and welcome to the PowerShell forum! Don’t apologize for being a “noob” or “newbie” or “n00b.” There’s just no need – nobody will think you’re stupid, and the forums are all about asking questions. Just ask! Use a descriptive subject. Don’t say “Need help” or “PowerShell Help”, actually summarize what the problem is. It helps the rest of us keep track of which problem is which. Don’t post massive scripts. We’re all volunteers and we don’t have time to read all that, nor will we copy…

mike9019 · 2021-04-24T11:22:37.000Z

Your comparison operators won’t work as you think they will if the left side of the comparison is already an array. Meaning in this code;

 if ("$pool " -notin $clients[$client].pool)

if $pool in this case is another array it’s best to use compare object or another method to look at all items in left array to compare to right arrays.

However, in your case if $pool is a single value, you’re having problems with comparison because you’re double quoting $pool and adding an additional space.

say $pool = ‘1’ when you initialize it. Now when you go to compare it, because of the double quotes, you’re actually saying…

if(1<space> -notin...)

Here’s an example of the items described above.

$DebugPreference = "Continue"
$pools = '123','456','789','100'
$poolString = '123'
$poolArray = '123','789'

Write-Debug "Test 1 - Double quotes around variable in comparison."
#Won't match because of the extra space in our quotes for $poolString
if("$poolString " -notin $pools) {
    "Didn't find `$poolString value '$poolString ' in `$pools $pools"
} else {
    "Found `$poolString value $poolString in `$pools $pools"
}

Write-Debug "Test 2 - No quotes around variable in comparison"
#Will match because we removed the double quotes and it's a single item
if($poolString -notin $pools) {
    "Didn't find `$poolString value '$poolString' in `$pools $pools"
} else {
    "Found `$poolString value $poolString in `$pools $pools"
}

Write-Debug "Test 3 - Comparing two array types with '-notin'."
if($poolArray -notin $poolArray){
    "Didn't find `$poolArray value(s) $poolArray in `$pools $pools"
} else {
    "Found `$poolArray value $poolArray in `$pools $pools"
}

And the output…

DEBUG: Test 1 - Double quotes around variable in comparison.     
Didn't find $poolString value '123 ' in $pools 123 456 789 100   
DEBUG: Test 2 - No quotes around variable in comparison
Found $poolString value 123 in $pools 123 456 789 100
DEBUG: Test 3 - Comparing two array types with '-notin'.
Didn't find $poolArray value(s) 123 789 in $pools 123 456 789 100

Here’s an example using your if statements but comparing array types differently…

$DebugPreference = "Continue"
$clients = @{
    Pepsi = @{
        db = '123'
        dd = '123'
        pool = '123','456','789'
    }
    Coke = @{
        db = '123'
        dd = '123'
        pool = '789','456','123'
    }
}

$client = 'RC'

$pool = '101','102','123' #works with a single item as well.
$db = '123','456'
$dd = '456','789'

if($clients[$client]) {
    #found a client match - check their $pool $db and $dd values for updates/changes?
    $poolCheck = $pool | Where-Object -FilterScript { $_ -notin $clients[$client]['pool']}
    if($poolcheck.Count -gt 0){
        $clients[$client]['pool'] += $poolCheck
    } else {
        "Nothing to add to pool"
    }

    #Do the same if statement for $dd and $db as you did for $pool...

} else {
    #no client, create new..
    $newClientData = @{
        db = [array]$db #even if they are single objects now, we want them to be arrays for later.
        dd = [array]$dd #even if they are single objects now, we want them to be arrays for later.
        pool = [array]$pool #even if they are single objects now, we want them to be arrays for later.
    }
    $clients.Add($client,$newClientData)
}

Write-Debug "RC"
$clients.RC
Write-Debug "Coke"
$clients.Coke
Write-Debug "Pepsi"
$clients.Pepsi

Output…

DEBUG: RC

Name                           Value
----                           -----
db                             {123, 456}
pool                           {101, 102, 123}
dd                             {456, 789}
DEBUG: Coke
db                             123                                                                                                                                                            
pool                           {789, 456, 123}
dd                             123
DEBUG: Pepsi
db                             123                                                                                                                                                            
pool                           {123, 456, 789}
dd                             123

If I change $client to ‘Coke’ so we can test the array of $pool against $clients.coke.pool’s current values here’s the output…

DEBUG: RC
DEBUG: Coke

Name                           Value
----                           -----
db                             123
pool                           {789, 456, 123, 101...}
dd                             123
DEBUG: Pepsi
db                             123                                                                                                                                                            
pool                           {123, 456, 789}
dd                             123

And the entire array of $clients.coke.pool…

PS C:\Users\Mike In IT> $clients.coke.pool

789
456
123
101
102

spiceuser-evezb · 2021-04-25T15:09:12.000Z

I have attached code here.

here input fil dups.txt

MN_INCR hup00013.corpads.local RMAN
MN_DR edimna800pf.corpads.local File
MN_WFULL edimna800pf.corpads.local File
MN_DR hup00013.corpads.local RMAN
MN_DR hup00009.corpads.local RMAN
MN_INCR hup00009.corpads.local RMAN

and dd_pool.txt

JC_Bootstrap JCBDD1300P
JC_INCR JCBDD1300P
DR_JC_Bootstrap MNBDD3302P
MN_INCR MNBDD3302P
JC_DR MNBDD3302P
JC_INCR_FS JCBDD1300P
MN_DR JCBDD1300P
MN_INCR_FS MNBDD3302P
JC_CloudTier_01 JCBDD1300P
JC_DR_DD2 MNBDD5304P

ouput file

=========

rsdbpc03.corpads.local,MN_DR MN_INCR_FS MN_DR MN_INCR ,File DB2 DB2 ,JCBDD1300P MNBDD3302P JCBDD1300P MNBDD3302P
ORAMNA300P.corpads.local,MN_DR MN_WFULL MN_CloudTier_01 MN_MFULL MN_INCR_FS ,File ,JCBDD1300P MNBDD3302P MNBDD3302P MNBDD3302P MNBDD3302P
rspwps03a.corpads.local,JC_DR JC_WFULL JC_CloudTier_01 JC_MFULL JC_INCR_FS ,File ,MNBDD3302P JCBDD1300P JCBDD1300P JCBDD1300P JCBDD1300P
nwnppl300p.corpads.local,DR_JC_Bootstrap JC_Bootstrap JC_CloudTier_01 ,File ,MNBDD3302P JCBDD1300P JCBDD1300P

code.txt (3.8 KB)

spiceuser-evezb · 2021-04-25T15:12:12.000Z

if u see pools MN_DR MN_INCR_FS MN_DR MN_INCR , MN_DR pool repeated and in db File DB2 DB2 , DB2 repeated , and in DD JCBDD1300P MNBDD3302P JCBDD1300P MNBDD3302P , dd they also repeated …

mike9019 · 2021-04-25T15:16:29.000Z

Was there something you didn’t understand from my first reply? Your duplication issue a d a solution is shown there.

spiceuser-evezb · 2021-04-25T15:40:15.000Z

I did tried below code

if($clients[$client]) {
#found a client match - check their $pool $db and $dd values for updates/changes?
$poolCheck = $pool | Where-Object -FilterScript { $_ -notin $clients[$client][‘pool’]}
if($poolcheck.Count -gt 0){
$clients[$client][‘pool’] += $poolCheck
} else {
“Nothing to add to pool”
}
#Do the same if statement for $dd and $db as you did for $pool…
} else {
#no client, create new…
$newClientData = @{
db = [array]$db #even if they are single objects now, we want them to be arrays for later.
dd = [array]$dd #even if they are single objects now, we want them to be arrays for later.
pool = [array]$pool #even if they are single objects now, we want them to be arrays for later.
}
$clients.Add($client,$newClientData)
}

and also tried blow peice

$poolCheck = $pool | Where-Object -FilterScript { $_ -notin $clients[$client][‘pool’]}
if($poolcheck.Count -gt 0){
$clients[$client][‘pool’] += $poolCheck
} else {
“Nothing to add to pool”
}
$dbCheck = $db | Where-Object -FilterScript { $_ -notin $clients[$client][‘db’]}
if($dbcheck.Count -gt 0){
$clients[$client][‘db’] += $dbCheck
} else {
“Nothing to add to db”
}
$ddCheck = $pool | Where-Object -FilterScript { $_ -notin $clients[$client][‘dd’]}
if($ddcheck.Count -gt 0){
$clients[$client][‘dd’] += $ddCheck
} else {
“Nothing to add to dd”
}

still getting duplicates …

can you modify my script and let me know ??

Thank you for helping.

mike9019 · 2021-04-26T09:47:14.000Z

Change your double quotes in all the .split() methods to single quotes see if that does what you need.

Debugging is your friend;

$DebugPreference = 'Continue'
$lines = Get-Content "$PSScriptRoot\dups.txt"
$ddlines = Get-Content  "$PSScriptRoot\pool_dd.txt"
$outFile = 'output.txt'
$clients = @{}
$dds = @{}
### get report
Write-Debug "what is `$clients before loops"
$clients
Write-Debug "==>`$clients"
foreach ( $line in $ddlines) {
    #split each line by single space - does not return anything in the split that is null/empty (multiple spaces)
    $splitDDline = $line.split(' ',[System.StringSplitOptions]::RemoveEmptyEntries)
    $pool = $splitDDline[0]
    $dd = $splitDDline[1]
    $vars = @"
  `n
    `$line = $line
    `$pool = $pool
    `$dd = $dd
"@
    Write-Debug $vars
    if ($pool -notin $dds.keys) {
        $dds[$pool] = $dd
    }
}

foreach ($line in $lines) {
    #split each line by single space - does not return anything in the split that is null/empty (multiple spaces)
    $splitDupsLine = $line.split(' ', [System.StringSplitOptions]::RemoveEmptyEntries)
    #set variables based on result of $splitLines
    $pool = $splitDupsLine[0]
    $client = $splitDupsLine[1]
    $db = $splitDupsLine[2]
    $dd = $dds[$pool]
    $varCheck = @"
    `n
    `$pool is $pool
    `$client is $client
    `$db is $db
    `$dd is $dd
    `$dds[`$pool] is $dds[$pool]
"@
    Write-Debug $varCheck
    if ($client -notin $clients.Keys) {
        Write-Debug "$client is not in `$clients.Keys"
        $clients[$client] = @{
            'pool' = @()
        }
        $clients[$client]['db'] = @{
            'db' = @()
        }
        $clients[$client]['dd'] = @{
            'dd' = @()
        }
        $clients[$client].pool = $pool
        $clients[$client].db = $db
        $clients[$client].dd = $dd
    }
    else {
        Write-Debug "$client found in `$clients.Keys"
        $poolCheck = $pool | Where-Object -FilterScript { $_ -notin $clients[$client]['pool'] }
        Write-Debug "`$poolCheck is $poolCheck"
        if ($poolcheck.Count -gt 0) {
            $clients[$client]['pool'] += $poolCheck
        }
        else {
            "Nothing to add to pool"
        }
        $dbCheck = $db | Where-Object -FilterScript { $_ -notin $clients[$client]['db'] }
        if ($dbcheck.Count -gt 0) {
            $clients[$client]['db'] += $dbCheck
        }
        else {
            "Nothing to add to db"
        }
        $ddCheck = $pool | Where-Object -FilterScript { $_ -notin $clients[$client]['dd'] }
        if ($ddcheck.Count -gt 0) {
            $clients[$client]['dd'] += $ddCheck
        }
        else {
            "Nothing to add to dd"
        }                                                    
                                                        
    }
}
Write-Debug "What is `$clients after loops"
$clients
$clients.GetEnumerator() | Sort-Object | ForEach-Object {
    "{0},{1},{2},{3}" -f ( $_.Name , $_.Value.pool, $_.Value.db, $_.Value.dd) | Out-File -FilePath $outFile -Append
    
}

spiceuser-evezb · 2021-04-27T11:32:12.000Z

Hi Mike

I used ur script and here are few lines from output file , if you notice MN_DR repeated and DD value mixed with pool.

rsdbpc03.corpads.local,MN_DRMN_INCR_FSMN_DRMN_INCR,FileDB2DB2,JCBDD1300PMN_INCR_FSMN_DRMN_INCR
ORAMNA300P.corpads.local,MN_DRMN_WFULLMN_CloudTier_01MN_MFULLMN_INCR_FS,File,JCBDD1300PMN_WFULLMN_CloudTier_01MN_MFULLMN_INCR_FS
rspwps03a.corpads.local,JC_DRJC_WFULLJC_CloudTier_01JC_MFULLJC_INCR_FS,File,MNBDD3302PJC_WFULLJC_CloudTier_01JC_MFULLJC_INCR_FS
nwnppl300p.corpads.local,DR_JC_BootstrapJC_BootstrapJC_CloudTier_01,File,MNBDD3302PJC_BootstrapJC_CloudTier_01
bpimna300p.corpads.local,MN_DRMN_WFULLMN_CloudTier_01MN_MFULLMN_INCR_FS,File,JCBDD1300PMN_WFULLMN_CloudTier_01MN_MFULLMN_INCR_FS

I have copied entire pool_dd here for you and attached pools , can you run at ur end and see difference ? zip_input has both file dups and pool_dd

Thanks again

JC_Bootstrap JCBDD1300P
JC_INCR JCBDD1300P
DR_JC_Bootstrap MNBDD3302P
MN_INCR MNBDD3302P
JC_DR MNBDD3302P
JC_INCR_FS JCBDD1300P
MN_DR JCBDD1300P
MN_INCR_FS MNBDD3302P
JC_CloudTier_01 JCBDD1300P
JC_DR_DD2 MNBDD5304P
JC_INCR_FS_DD2 JCBDD2301P
MN_CloudTier_01 MNBDD3302P
MN_WFULL MNBDD3302P
MN_MFULL MNBDD3302P
JC_WFULL JCBDD1300P
JC_WFULL_DD2 JCBDD2301P
JCBDD1300Pcorpadslocal JCBDD1300P
JC_MFULL JCBDD1300P
DR_MN_DB_MO JCBDD1300P
JCBDD_AIX_IP JCBDD1300P
DR_JC_DB MNBDD3302P
JC_INCR_FC_09 JCBDD1300P
JC_DB_INCR JCBDD1300P
JC_UNIX_MO JCBDD1300P
JC_UNIX_FC_MO JCBDD1300P
MN_DB_MO MNBDD3302P
CTTEST_01 MNBDD3302P
JC_UNIX MNBDD3302P
JCBDD_AIX_FC MNBDD3302P
JC_MFULL_DD2 JCBDD2301P
JC_DB JCBDD1300P
JC_UNIX_FC JCBDD1300P
MN_UNIX MNBDD3302P
JC_DB_MO JCBDD1300P
DD_0129 JCBDD1300P
MN_UNIX_MO MNBDD3302P
CTEST_03 JCBDD1300P
MN_DB MNBDD3302P
JC_CloudTier_DD2_01 JCBDD2301P
DR_JC_UNIX_MO MNBDD3302P
DR_JC_UNIX_FC MNBDD3302P
DR_MN_UNIX_MO JCBDD1300P
DR_MN_DB JCBDD1300P
cloud_test_09 JCBDD1300P
CT_JC_DB_MO JCBDD1300P
DR_MN_UNIX JCBDD1300P
DR_JC_DB_MO MNBDD3302P
JC_CLOUD_RECALL JCBDD1300P
DR_JC_UNIX MNBDD3302P
DR_JC_UNIX_FC_MO MNBDD3302P
Cloud_E_0129 JCBDD1300P

dups.txt (41.7 KB)

mike9019 · 2021-04-27T12:59:31.000Z

spiceuser-evezb:

Hi Mike

I used ur script and here are few lines from output file , if you notice MN_DR repeated and DD value mixed with pool.

rsdbpc03.corpads.local,MN_DRMN_INCR_FSMN_DRMN_INCR,FileDB2DB2,JCBDD1300PMN_INCR_FSMN_DRMN_INCR
ORAMNA300P.corpads.local,MN_DRMN_WFULLMN_CloudTier_01MN_MFULLMN_INCR_FS,File,JCBDD1300PMN_WFULLMN_CloudTier_01MN_MFULLMN_INCR_FS

MN_DR repeated because for ‘client’ rsdbpc03.corpads.local the MN_DR ‘pool’ has two different DBs - “file” and “DB2” - script is working as designed.

DD value mixed with pool - it didn’t mix - this is a result of your $clients.GetEnumerator() string formatting code.

Your original request was to prevent duplicates and the suggestions made here did/does exactly that.

If the suggestions are not providing you with the output or results you’re expecting please detail what it is you’re attempting to do and explain (describe) your data set(s). Submitting files to look at does not help me understand your end goal or what the data is and how the data relates.

spiceuser-evezb · 2021-04-27T15:20:48.000Z

Mike

I just need one line for every client , in that line first entry should be client , then pool ( no duplicate pool ) , then db ( again no duplicate ) then line ends with DD ( again no duplicate ). I have file( pool_dd) which has line pool to DD relation ( here no duplicate entry). dups file has pool , client and db value, here we have client repeated with different pool and db values.

Thanks again

mike9019 · 2021-04-27T15:47:51.000Z

What do you consider a duplicate - which value is supposed to be unique between your data sets; the client, pool, dd, and/or db?

Here’s an example of the most recent files you sent for dups and pool_dd;

pool_dd.txt has a DD named “JCBDD1300P” 26 times - each of those with a unique ‘pool’ value.

dups.txt has a client named “bpdppa300p.corpads.local”. Taking the dups.txt pool value of ‘JC_CloudTier_01’ against that client it shows two different db (File, DB2).

What would you expect your end result to be with this example?

The way in which the current script works you should get at least two results;

client,pool,db,dd
bpdppa300p.corpads.local, JC_CloudTier_01,File, JCBDD1300P
bpdppa300p.corpads.local, JC_CloudTier_01,DB2, JCBDD1300P

Are these considered duplicates since client, pool, and DD match? The DB differs so not a duplicate to me. This is what I mean by describing your data and explaining in detail your goal.

spiceuser-evezb · 2021-04-27T15:57:59.000Z

If you see below line from output file…

rsdbpc03.corpads.local,MN_DR MN_INCR_FS MN_DR MN_INCR ,File DB2 DB2 ,JCBDD1300P MNBDD3302P JCBDD1300P MNBDD3302P

pools are MN_DR MN_INCR_FS MN_DR MN_INCR , here MN_DR came two times

db are File DB2 DB2 , here DB2 came two times

dd are JCBDD1300P MNBDD3302P JCBDD1300P MNBDD3302P , here JCBDD1300P came out two times same for MNBDD3302P

So i want ouput look like

rsdbpc03.corpads.local,MN_DR MN_INCR_FS MN_INCR ,File DB2,JCBDD1300P MNBDD3302P

mike9019 · 2021-04-27T16:20:52.000Z

Much better explanation!

Your current issues can be explained by the data coming in. I took your dups and pool files and imported into Excel. Using PowerQuery I did a left inner join on pool.pool = dups.pool.

Filtering on client ‘rsdbpc03.corpads.local’ you can see why;

MN_DR came in twice, because there’s two results for MN_DR pool that have two different DBs.

DB2 came in twice because there are two different pools with the same DB.

JCBDD1300P and MNBDD3302P came in twice because each of those have two different pools and DBs.

Now that you can visualize the data maybe you can try and come up with a solution

spiceuser-evezb · 2021-04-27T16:46:37.000Z

Let me ask you if we parse output file line by line , read pool info and remove reoccurrence and do same for db an dd ?

Using split by , ( comma ) then parse each output and look for duplicates ? then put together in line . is this duable ?

Now my original question why array in hash table appending same value when i used “notin” match operation. If i am appending space to value , it should take entire thing and do look up ( adding space so each one is separated by space ).

Thank you

mike9019 · 2021-04-27T18:11:35.000Z

spiceuser-evezb:

Let me ask you if we parse output file line by line , read pool info and remove reoccurrence and do same for db an dd ?

Using split by , ( comma ) then parse each output and look for duplicates ? then put together in line . is this duable ?

Try it - see what happens I prefer to manipulate the data before sending it out then reading it back in but if it gets you where you need to be there’s no real issue that I can tell besides a possible performance hit. May or may not be relevant depending on the size of your data sets.

Now my original question why array in hash table appending same value when i used “notin” match operation. If i am appending space to value , it should take entire thing and do look up ( adding space so each one is separated by space ).

Thank you

This was explained here: array in hash table having duplicates