Use Python to Query a LARGE SharePoint list.

When querying a SharePoint list that has more than 5,000 items, you’ll likely receive an error like this:

This view cannot be displayed because it exceeds the list view threshold (5000 items) enforced by the administrator. 


Microsoft.SharePoint.SPQueryThrottledException', 'The attempted operation is prohibited because it exceeds the list view threshold.', "500 Server Error: Internal Server Error for url

Or, your query will only return the default 100 items. To get around this, pagination can be used to query the list and return ALL of the items.
Example:
all_items = list_to_export.items.paged(1000).get().execute_query()

Full script using VS Code:

from office365.runtime.auth.authentication_context import AuthenticationContext
from office365.sharepoint.client_context import ClientContext


app_settings = {
    'url': 'https://taco.sharepoint.com/sites/queso/',
    'client_id': 'ID here',
    'client_secret': 'shhhh its a secret',
}

context_auth = AuthenticationContext(url=app_settings['url'])
context_auth.acquire_token_for_app(client_id=app_settings['client_id'], client_secret=app_settings['client_secret'])

#connect to the site
ctx = ClientContext(app_settings['url'], context_auth)
ctx.execute_query()

#get the target list
list_title = "List of Tacos"
list_to_export = ctx.web.lists.get_by_title(list_title)

#get all of the list items
all_items = list_to_export.items.paged(1000).get().execute_query()
list_items = [item for item in all_items]

print("Item count: {0}".format(len(list_items)))

The example above connects to a SharePoint site using a client ID and secret, then queries the list. Again, the key here is using pagination (paged). You can adjust the page size to better fit your needs, but be sure to leave it under 5,000, or you will be back to square one.

Flow and SharePoint Online Large Libraries

Using Flow to get or check for files in a large SharePoint library can be a little tricky. If you are sure your library will always stay under 5,000 items the Get Files (properties only) Flow action is a quick n’ simple approach to use. When your library crosses over the mythical 5k mark or somewhere in that neighborhood, the Get Files action will fail to return results without warning. What I’m outlining below are other options when working with large libraries.

Setup:



Option 1: Get Files using an Odata filter query
downside: only use this in small libraries



Result:


Option 2: use the SharePoint API
downside: the lack of transparency from Microsoft related to how often data is crawled.

Result:

Option 3: use the SharePoint API along with a filter action on the library. This option does require that you have metadata set up on the library to filter on and there is not a wildcard / contains option. The filtered value needs to be exact.
downside: you will need to set up your metadata ahead of time or create it after the fact then backfill.

Result:

The more I learn about Flow and SharePoint Online, the more I’m starting to like Option 3 when doing a lookup type of search. SharePoint Search is an extremely powerful tool if the search index is fresh.

Flow Variables Not Displaying In The Dynamic Content Menu

I was trying to dynamically set a SharePoint list item ID to a variable but the variable was not displaying in the Dynamic Content menu.

The solution was to use an expression to set the value, save the flow, exit the edit screen, then re-enter the edit screen.

variables('varRefID')
Create a variable
Notice the variable is not available for selection
Use the expression window to enter the variable name
The variable has been added but looks weird. This is not an issue but continue reading if you want to clean display value.
Save your flow and exit the edit screen.
Edit your flow
The variable is now displayed correctly!

Copy Files From Azure File Storage to SharePoint

As of today, there is not a Logic App trigger for Azure File Storage, so I went with a schedule-based approach. Yes, this example leaves out a lot of fine-tuning, but it will get you headed in the right direction.

Create a blank Logic app
Trigger: Schedule
Action: Azure File Storage – List files
Action: SharePoint – Create file

After you add the SharePoint action, the Logic App should automatically add a For Each action and place the SharePoint Create File action inside of it.

Overview of the Logic App
For each action expanded
Testing the Logic App

In the last screenshot, I tested the Logic App by uploading a couple of documents in Azure Storage Explorer, then I manually ran the Logic App (click the Run button).

Again, this is a simple example. The example does not account for processing the same files over and over…

Power Automate Bad Gateway Error

I was trying to use a SQL Insert Row action to insert a new row in a SQL Server table and received a Bad Gateway error. First, I thought it was a permissions issue, then I thought my Flow stopped working…

Turned out to be an issue with the amount of data being inserted into a field. One SQL column was set to varchar(X) and the Flow was trying to insert more characters than X.

System.MissingMethodException: Method not found Connect-PnPOnline

Using Visual Studio Code and SharePoint PNP I was trying to make some updates to a list but I wasn’t able to connect to a site.

Connect-PnPOnline -Url "https://taco.sharepoint.com/" -Credentials $creds

Error I was receiving:
System.MissingMethodException: Method not found: ‘System.Runtime.Remoting.ObjectHandle System.Activator.CreateInstance(System.String, System.String)’. at SharePointPnP.PowerShell.Commands.Base.ConnectOnline.ProcessRecord() at System.Management.Automation.CommandProcessor.ProcessRecord()

I tried uninstalling VScode, removed all traces of SharePoint from my laptop, and cleared the GAC. Nothing worked.

Here is what did work:
In VScode:

  1. Open the Command Palette on Windows or Linux with Ctrl+Shift+P. On macOS, use Cmd+Shift+P.
  2. Search for Session.
  3. Click on PowerShell: Show Session Menu.
  4. Choose one of the ___ (x86) options

Not sure how, but I was using an x64 session and SharePoint PNP clearly didn’t like that.

Edit: Updated VScode to the latest version and it managed to reset my session settings. When this happened, it caused my CSOM scripts to report a The remote server returned an error: (400) Bad Request error. The fix above will resolve the error.

How to use an iFrame in a modern SharePoint Online page

Using the Embed web part I was trying to paste in a site URL when I should have been using the iFrame HTML tag.

example:

<iframe src="https://sharepointed.com" height="200" width="300"></iframe>

If you encounter this error: This website doesn’t support embedding using just the address ….

You will need to update the HTML Field Security settings in the Site Settings area of your site. In my case, I simply added sharepointed.com to the allow iframes from this domain list, then updated the web part again.

Connect to SharePoint Online Using PowerShell

Update and a much better way to approach this:
Use a SharePoint App Only Client Id and Secret to access the site, list, or library.

Microsoft documentation:
https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/security-apponly-azureacs
You can create an app principle that is limited to a single site, list, library, or a combination of them:
https://piyushksingh.com/2018/12/26/register-app-in-sharepoint/

 $sampleConnect = Connect-PnPOnline -Url "https://YOURsite.sharepoint.com/sites/parent/child" -AppId "12345-94c3-4149-bda5-abcedffadsf" -AppSecret "643r4er5sfdadsfadsfdsf=" -ReturnConnection

Write-Host  $sampleConnect.Url
In this example, I’m connecting to a Site Collection on my tenant.

Assumptions:
1) You have created a token in your o365 site
1.1) https://portal.office.com/account/
1.2) On the left site of the page click Security & privacy, then click Create and manage app passwords
1.3) In the app password page click the create button and give it a name.
1.4) Save the password to a secure location.
1.5) There is a better way of doing this that I will cover in a future post.
2) You have downloaded to CSOM DLL(s) from Nuget

Clear-Host

$userName = "me@sharepointed.com"
$pw = "abc123taco"  # I"M USING AN APP PASSWORD 
$siteCollectionUrl = "https://sharepointed.sharepoint.com/sites/taco"

#Secure the password
$securePassword = ConvertTo-SecureString $pw -AsPlainText -Force

Add-Type -Path "C:\Code\DLL\Microsoft.SharePoint.Client.dll"
Add-Type -Path "C:\Code\DLL\Microsoft.SharePoint.Client.Runtime.dll"

#Create Context
$ctx = New-Object Microsoft.SharePoint.Client.ClientContext($siteCollectionUrl)

#Authorise
$ctx.Credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($userName, $securePassword)

$web = $ctx.Web
$properties = $web.AllProperties
$ctx.Load($web)
$ctx.Load($properties)
$ctx.ExecuteQuery()

Write-Host " Site Collectione URL: $($web.Url)"
Write-Host "Properties are "

foreach ($prop in $properties) {
    $prop.FieldValues
}