- Monitoring Microsoft 365 with SCOM and the NiCE Active 365 Management Pack - Tue, Feb 7 2023
- SCOM.Addons.MailIn: Monitor anything that can send email with SCOM - Mon, May 25 2020
- Display a user’s logged-on computer in Active Directory Users and Computers (ADUC) - Mon, Jan 21 2019
It may happen because we sized a disk wrongly, did not expect growth of the WinSxS folder, neglected to stop IIS logging during troubleshooting, or because the developer did not implement a log-cleanup routine in her application. Full disks can have many reasons.
The first step when receiving an alert about a filling disk is to RDP into a server and use tools to analyze the disk consumption. This takes time and reoccurs—an ideal candidate for scripting and automation.
Diagnostic tasks in SCOM can run scripts or commands directly on the affected machine when an alert occurs.
Preparing SCOM
By default, only VBScript can create diagnostic and recovery tasks. Download and install a free, open-source PowerShell community management pack on GitHub to use PowerShell. You likely have already imported management packs for Windows Server operating systems.
PowerShell script
The PowerShell script below gathers disk usage. Download the script and name it Get-LargeDirectoriesAndFiles.ps1.
param([string]$Arguments) $startDirectory = $Arguments + '\' $numberOfTopDirectories = 5 $numberOfTopFiles = 5 $emailTo = 'adminteam@contoso.msft' $smtpSrv = 'mailer.contsole.msft' $emailFrom = 'diskSpaceDetail@scom.contoso.msft' #region Get-Metadata $timeZone = ([TimeZoneInfo]::Local).Id $scanDate = Get-Date -Format 'yyyy-MM-dd hh:MM:ss' $WindowsVersion = Get-WmiObject -Class Win32_OperatingSystem | Select-Object -ExpandProperty Caption try { $computerDescription = Get-ItemProperty -Path HKLM:\SYSTEM\CurrentControlSet\services\LanmanServer\Parameters | Select-Object -ExpandProperty srvcomment } catch { $computerDescription = '' } try { $adSearcher = New-Object System.DirectoryServices.DirectorySearcher $adSearcher.Filter = "(&(objectCategory=computer)(cn=$env:computername))" $adComputer = $adSearcher.FindOne() $adComputerProperties = $adComputer | Select-Object -ExpandProperty Properties $adComputerLdapPath = $adComputerProperties.distinguishedname } catch { $adComputerLdapPath = 'Failed to extract AD Information.' + $_.StackTrace } $diskDetails = Get-WMIObject -Namespace root/cimv2 -Class Win32_LogicalDisk | Where-Object {$_.DeviceID -match "$($Arguments)" } | Select-Object -Property Size, FreeSpace, VolumeName, DeviceID $DriveLetter = $diskDetails.DeviceID $DriveName = $diskDetails.VolumeName $SizeInGB = "{0:0}" -f ($diskDetails.Size/1GB) $FreeSpaceInGB = "{0:0}" -f ($diskDetails.FreeSpace/1GB) $PercentFree = "{0:0}" -f (($diskDetails.FreeSpace / $diskDetails.Size) * 100) $diskMessage = "<table><tr><td>$($DriveLetter)</td><td> ($($DriveName))</td></tr>" $diskMessage += "<tr><td>Total: $($SizeInGB) GB |</td><td>Free: $($FreeSpaceInGB) GB ($($PercentFree) %)</td></tr></table>" #endregion Get-Metadata Function Get-BigDirectories { param( [string]$startDirectory, [ref]$sortedDirectories ) $bigDirList = New-Object -TypeName System.Collections.ArrayList if (Test-Path -Path $startDirectory) { & "$env:ComSpec" /c dir $startDirectory /-c /s | ForEach-Object { $null = $_ -match 'Directory\s{1}of\s{1}(?<dirName>[\w:\\\s\.\-\(\)_#{}\$\%\+\[\]]{1,})' $dirName = $Matches.dirName $null = $_ -match '\s{1,}\d{1,}\sFile\(s\)\s{1,}(?<lengh>\d{1,})' $dirLength = $Matches.lengh if ($dirName -and $dirLength) { $dirLength = [float]::Parse($dirLength) $myFileHsh = @{'Name'=([string]$dirName)} $myFileHsh.Add('Length',$dirLength) $myFileObj = New-Object -TypeName PSObject -Property $myFileHsh $null = $bigDirList.Add($myFileObj) } $dirName = '' $dirLength = 0 } #END cmd /c dir C:\windows /-c /s | ForEach-Object } else { if ($startDirectory) { $dirName = 'Error' $dirLength = 'No directory passed in.' } else { $dirName = $startDirectory $dirLength = $error.Message.ToString() } $dirLength = [float]::Parse($dirLength) $myFileHsh = @{'Name'=([string]$dirName)} $myFileHsh.Add('Length',$dirLength) $myFileObj = New-Object -TypeName PSObject -Property $myFileHsh $null = $bigDirList.Add($myFileObj) } #END if (Test-Path -Path $startDirectory) $sortedDirectories.Value = $bigDirList } #End Function Get-BigDirectories Function Convert-LengthToReadable { param( [System.Collections.ArrayList]$lengthList, [ref]$readableList ) $allFiles = New-Object -TypeName System.Collections.ArrayList $lengthList | ForEach-Object { $sizeRaw = $_.Length $sizeUnit = 'KB' $fileSize = 0 if ($sizeRaw -gt 1kb -and $sizeRaw -lt 1mb ) { $fileSize = $sizeRaw / 1mb $sizeUnit = 'MB' } elseif ($sizeRaw -gt 1mb -and $sizeRaw -lt 1gb ) { $fileSize = $sizeRaw / 1mb $sizeUnit = 'MB' } elseif ($sizeRaw -gt 1gb -and $sizeRaw -lt 1tb ) { $fileSize = $sizeRaw / 1gb $sizeUnit = 'GB' } elseif ($sizeRaw -gt 1tb ) { $fileSize = $sizeRaw / 1tb $sizeUnit = 'TB' } else { $fileSize = $sizeRaw $sizeUnit = 'KB?' } $fileSize = [Math]::Round($fileSize,2) $myFileHsh = @{'Name'=([string]$_.Name)} $myFileHsh.Add('fileSize',([float]$fileSize)) $myFileHsh.Add('sizeUnit',([string]$sizeUnit)) $myFileHsh.Add('Length',([float]$sizeRaw)) $myFileHsh.Add('LastWriteTime',$_.LastWriteTime) $myFileObj = New-Object -TypeName PSObject -Property $myFileHsh $null = $allFiles.Add($myFileObj) } $readableList.Value = $allFiles } #End Function Convert-LengthToReadable Function Send-TopDirectoryMailReport { param( [string]$diskMetaData, [string]$runInfo, [System.Array]$tDirectories, [System.Collections.Hashtable]$tDirsAndFiles, [System.Collections.Hashtable]$nDirsAndFiles ) $directoryDetails = '' $dirAndFilesDetails = '' $dirAndNewFilesDetails = '' foreach ($dirItem in $tDirectories) { $dirAndFilesDetails += "<tr><td style=`"font-weight: bold; text-align:center`"><br /> $($dirItem.Name)</td></tr>" $dirAndNewFilesDetails += "<tr><td style=`"font-weight: bold; text-align:center`"><br /> $($dirItem.Name)</td></tr>" $directoryDetails += "<tr><td>$($dirItem.Name)</td><td>$($dirItem.fileSize)</td><td>$($dirItem.sizeUnit)</td><tr>" $matchEntry = $tDirsAndFiles.($dirItem.Name) $matchEntry | ForEach-Object { $dirAndFilesDetails += "<tr><td>$($_.Name)</td><td>$($_.fileSize)</td><td>$($_.sizeUnit)</td><td>$($_.LastWriteTime)</td><tr>" } $matchEntry = $nDirsAndFiles.($dirItem.Name) $matchEntry | ForEach-Object { $dirAndNewFilesDetails += "<tr><td>$($_.Name)</td><td>$($_.fileSize)</td><td>$($_.sizeUnit)</td><td>$($_.LastWriteTime)</td><tr>" } } #End foreach ($dirItem in $tDirectories) $htmlBegin = "<!DOCTYPE html><html><head><title>DISK FULL - Troubleshooting Assistance on $($env:Computername) - $($computerDescription)</title>" $htmlBegin += "<h1><span style=`"background-color:#D3D3D3`">DISK FULL - Troubleshooting Assistance on $($env:Computername)</span></h1></head>" $htmlBegin += "<h3><span style=`"background-color:#D3D3D3`"> $($computerDescription) </span></h3></head>" $htmlMiddle = '<body style="color:#000000; font-size:12pt;"><p> </p><span style="color:#B22222; font-weight: bold; background-color:#D3D3D3; font-size:14pt;">Disk details:</span><br />' + $diskMetaData + '<p> </p>' $htmlMiddle += '<span style="color:#000080; font-weight: bold; background-color:#D3D3D3; font-size:14pt;">Largest Directories:</span><br /><br /> <table>' + $directoryDetails + '</table><p> </p>' $htmlMiddle += '<span style="color:#000080; font-weight: bold; background-color:#D3D3D3; font-size:14pt;">Newest files:</span><br /><table>' + $dirAndNewFilesDetails + '</table><p> </p>' $htmlMiddle += '<span style="color:#000080; font-weight: bold; background-color:#D3D3D3; font-size:14pt;">Largest files:</span><br /><table>' + $dirAndFilesDetails + '</table><p> </p>' $htmlMiddle += '<span style="color:#FF8C00; font-weight: bold; background-color:#D3D3D3; font-size:14pt;">Meta Information:<br /></span>' + $runInfo $htmlEnd = '</body></html>' $htmlCode = $htmlBegin + $htmlMiddle + $htmlEnd $mailMessageParms = @{ To = $emailTo From = $emailFrom Subject = "DISK Full - Troubleshooting Assistance on $($env:computername) . $($computerDescription)" Body = $htmlCode Smtpserver = $smtpSrv ErrorAction = "SilentlyContinue" BodyAsHTML = $true } Send-MailMessage @mailMessageParms } #End Send-TopDirectoryMailReport $startTime = Get-Date $bigDirList = New-Object -TypeName System.Collections.ArrayList Get-BigDirectories -startDirectory $startDirectory -sortedDirectories ([ref]$bigDirList) $bigDirListReadable = New-Object -TypeName System.Collections.ArrayList Convert-LengthToReadable -lengthList $bigDirList -readableList ([ref]$bigDirListReadable) $topDirectories = $bigDirListReadable | Sort-Object -Property Length -Descending | Select-Object -First $numberOfTopDirectories $topDirsAndFiles = New-Object -TypeName System.Collections.Hashtable $topDirsAndNewestFiles = New-Object -TypeName System.Collections.Hashtable foreach ($tDirectory in $topDirectories) { $tDirName = $tDirectory.Name $tmpFileList = New-Object -TypeName System.Collections.ArrayList $tmpNewFileList = New-Object -TypeName System.Collections.ArrayList $newFilesInDir = New-Object -TypeName System.Collections.ArrayList $filesInTDir = Get-ChildItem -Path $tDirName | Where-Object { $_.PSIsContainer -eq $false } | Select-Object -Property DirectoryName, Name, LastWriteTime, Length $filesInTDir | ForEach-Object { $null = $newFilesInDir.Add($_) } $filesInTDir = $filesInTDir | Sort-Object -Property Length -Descending | Select-Object -First $numberOfTopFiles $newFilesInDir = $newFilesInDir | Sort-Object -Property LastWriteTime -Descending | Select-Object -First $numberOfTopFiles $filesInTDir | Select-Object -Property DirectoryName, Name, LastWriteTime, Length | ForEach-Object { $null = $tmpFileList.Add($_) } $newFilesInDir | Select-Object -Property DirectoryName, Name, LastWriteTime, Length | ForEach-Object { $null = $tmpNewFileList.Add($_) } $bigFileList = New-Object -TypeName System.Collections.ArrayList Convert-LengthToReadable -lengthList $tmpFileList -readableList ([ref]$bigFileList) $newFileList = New-Object -TypeName System.Collections.ArrayList Convert-LengthToReadable -lengthList $tmpNewFileList -readableList ([ref]$newFileList) $topDirsAndFiles.Add($tDirName,$bigFileList) $topDirsAndNewestFiles.Add($tDirName,$newFileList) } #End foreach ($tDirectory in $topDirectories) $endTime = Get-Date $requiredTime = New-TimeSpan -Start $startTime -End $endTime $reqHours = [Math]::Round($requiredTime.TotalHours,2) $reqMinutes = [Math]::Round($requiredTime.TotalMinutes,2) $reqSeconds = [Math]::Round($requiredTime.TotalSeconds,2) $metaInfo = "<table><tr><td>Gathering details took:</td><td>$($reqHours) Hours / $($reqMinutes) Minutes / $($reqSeconds) Seconds.</td></tr>" $metaInfo += "<tr><td>Checking time:</td><td>$($scanDate), $($timeZone)</td></tr>" $metaInfo += "<tr><td>LDAP Path:</td><td>$($adComputerLdapPath)</td></tr>" $metaInfo += "<tr><td>Operating System:</td><td>$($WindowsVersion)</td></tr></table>" $sendTopDirectoryMailReport = @{ tDirectories = $topDirectories tDirsAndFiles = $topDirsAndFiles nDirsAndFiles = $topDirsAndNewestFiles diskMetaData = $diskMessage runInfo = $metaInfo } Send-TopDirectoryMailReport @sendTopDirectoryMailReport
Create the diagnostic task
In the SCOM console, switch to the Authoring pane and expand Management Pack Objects and Monitors. Limit the search to logical disk. Right click Windows [20XX] Logical Disk Free Space Monitor.
Switch to the Diagnostic and Recovery tab, click Add…, and choose Diagnostic for warning health state.
Chose Run a PowerShell script (Community) and click New… to create a management pack to store the task in.
Choose a fitting name like Windows.Server.Custom.Tasks and proceed by clicking Next.
As an optional step, you can specify additional information in the Knowledge section. Proceed by clicking Create.
Back in the Task Wizard, ensure that Run a PowerShell script (Community) is still selected and click Next.
In the General section, specify a task name, such as Disk Full - Troubleshooting Assistance and optionally a description, such as Scans the partition and sends a utilization report to the admin team. Proceed by clicking Next.
In the Script section, enter a File Name, such as Get-LargeDirectoriesAndFiles.ps1, set the timeout to 5 minutes, and paste the Script into it.
Set the variable $emailTo to specify the recipient of the disk usage report, such as adminteam@contoso.msft. The variable $smtpSrv specifies the name that will send the mail, and set $emailFrom specifies the sender address.
Optionally you can adjust the values for the number of directories ($numberOfTopDirectories) and files ($numberOfTopFiles) to show on the report.
Stay on the current screen and click Parameters. There, select Device Identifier (Windows Logical Hardware Component) from the list.
Initiate the task creation by clicking Create. Depending on your environment, this may take a while.
Repeat the steps for the Critical health state.
Repeat the procedure for all other Windows Server versions. Otherwise it will only work for Windows Server 2008.
The next time a disk fills up, you'll receive an e-mail like the one below.
Thank you for this wonderful tool. I was wondering the best way to scope this to only a certain group of servers. I have 2 different monitors set up. One for our Server group, and we get notified on all servers. I have another monitor set up for another group and they only want to be notified on certain servers.
I have tried disabling the monitor by default and then enabling it only for that group of servers, but I am getting this error:
Note: The following information was gathered when the operation was attempted. The information may appear cryptic but provides context for the error. The application will continue to run.
: Database error. MPInfra_p_ManagementPackInstall failed with exception:
Failed to validate item: Alias46bfc1e52d1c4f779d1cfdbae45c3117OverrideForDiagnosticMomUIGenaratedDiagnostic6d9af75d83184e4f9672592b78436789ForContextUINameSpace67f813fc09bb4862bbfe98ed5021c116Group
What is the best way to do what I am wanting to do?
Thank you for your reply.
Hi Emily,
I think keeping the recovery disabled by default and only enabling it for groups of servers is already the right approach.
In terms of the error message I am a bit puzzled. I haven’t meet it so far.
Could you describe the steps you’ve done? Like 1. 2. 3. ?
Have you tried it a second time?
Are there other alerts below the Operations Manager folder?
Ruben
Here are my steps:
1. I created the script as you described and chose not to run by default (I am only focused on the one marked No):
2. I then go to overrides, select the diagnostic task and choose override for a group. I select the group I have made and click OK
3. I then choose to enable the rule and click OK
Then the error pops up
I have tried it numerous times with the same results.
I am not sure what you are asking about other alerts below the Operations Manager folder.
Thank you for your time.
I tried replying this to the email I was sent as well, but not sure if it went through or not.
Hi Emily,
whenever I meet errors mentioning something about “Database xyz” then it was because either the OpsMgr DB or the OpsMgr DW was running out of disk space.
Could you please check if both databases have enough free space?
In the SCOM console, please also check in Monitoring section the “Operations Manager” folder. Perhaps there are some other interesting messages in the “Active Alerts” views.
If I understand you right, you unchecked “Run diagnostic automatically” in the step which is title ‘Name and describe the diagnostic task’? If so, you could try to keep it enabled and in the dropdown below where you select the target, you can choose a group of servers you like to have it enabled.- You would need to create a dynamic group first of cause.
… or, if all is not working just keep it as described above. If someone doesn’t like those e-mails they can create a filter-rule to move them directly into the recycle bin 😉
I really appreciate your help. I tried to keep it enabled and then select a target, but the box is blank and will not search for the groups I have created.
I also looked at the database and it has plenty of space and there aren’t any strange errors in the ops man alerts.
I think I will just leave it as is, and they can ignore the ones that aren’t important to them.
Thank you again for your help.
Thanks for the feedback, Emily. 🙂
Hi,
Just discovered this awesome report! I configured it according to the guide but Im not receiving any emails and Im not sure that the diagnostic task even fires. Do you have any ideas?
Hi,
The report works perfect but sections described below are empty:
Largest Directories:
Newest files:
Largest files:
What could it be?
Thanks a lot for your time
I have just detected the problem. I ejecute this in Spanish OS, so i had to make some changes in regex expressions.
Hi Ruben,
I have tried to run your powershell code, but note no data returned for the disk details:
Disk details:
C: E: (SYSTEM )
Total: GB | Free: GB ( %)
Method invocation failed because [System.Object[]] does not contain a method named ‘op_Division’.
At line:29 char:1
+ $SizeInGB = “{0:0}” -f ($diskDetails.Size/1GB)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (op_Division:String) [], RuntimeException
+ FullyQualifiedErrorId : MethodNotFound
Method invocation failed because [System.Object[]] does not contain a method named ‘op_Division’.
At line:30 char:1
+ $FreeSpaceInGB = “{0:0}” -f ($diskDetails.FreeSpace/1GB)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (op_Division:String) [], RuntimeException
+ FullyQualifiedErrorId : MethodNotFound
Method invocation failed because [System.Object[]] does not contain a method named ‘op_Division’.
At line:31 char:1
+ $PercentFree = “{0:0}” -f (($diskDetails.FreeSpace / $diskDetails. …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (op_Division:String) [], RuntimeException
+ FullyQualifiedErrorId : MethodNotFound
Apparently, you can’t both pipe and create a result variable.
https://social.technet.microsoft.com/Forums/en-US/73197c41-b6c3-489b-be3a-696acc1ae0df/method-invocation-failed-because-systemobject-does-not-contain-a-method-named?forum=winserverpowershell
$deviceID = ‘C:’
Get-WMIObject Win32_LogicalDisk -Filter “DeviceId=’$deviceID'” |
Select-Object Size, FreeSpace, VolumeName, DeviceID, @{n=’Size(Gb)’;e={[int]($_.Size/1GB)}}
Hope you can assist!?
How can you troubleshoot the execution of this Script, is there a location on the Client of Management Server that has logs for review?